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6 On Top of the Hierarchy: How 
Guidelines Shape Systematic 
Reviewing in Biomedicine 


Alexander Schniedermann, Clemens Bliimel, 
and Arno Simons 


Certifying Biomedical Evidence 


Evidence practices in biomedicine have changed profoundly since the postwar 
era. After medical treatments ceased to reveal “blockbuster” effects at the 
beginning of the twentieth century, experts developed and promoted more 
systematized attempts to determine the most effective “therapeutic inter- 
ventions.” They redefined the ways of how knowledge can be legitimately 
produced and claimed. In this manner, the randomized-controlled-trial 
(RCT) combined different methods and techniques and has become the 
main template for generating biomedical knowledge. Full of ethical and 
epistemic promises, proponents turned this research design into a marker for 
“evidence” in biomedical policy and practice, and one which allows for 
certificating knowledge claims irrespective of the various complexities in their 
making.” 

But not only experimental designs became part of the evidence movement 
in biomedicine. Also, research syntheses summarizing and synthesizing pri- 
mary research were highly demanded. By using systematic reviewing to find 
and appraise the relevant trials, as well as meta-analyses as a reliable statistical 
technique to pool trial data, researchers started to aggregate multiple trial 
results into overall conclusions about treatment effectivity.? Facing the 
growing output of the “clinical trial industry,”* experts hoped that these 
methods would help to cope with a huge load of information. Further, it was 
hoped that the aggregation of studies would solve contradictive results and 
sufficiently represent scientific consensus. These steps toward fact making 
would then ultimately allow for drawing conclusions and recommendations 
as to what knowledge should be perceived as “evidence” and which inter- 
ventions are to be considered effective.” 

Like clinical trials, systematic reviews were ascribed evidencing qualities 
based on the intended purpose of the genre as identifying and presenting 
the most convincing research. Displaying such authority claims, meta- 
analysis and systematic review have been coined the “platinum standard of 
evidence,” in comparison to RCT as the “the gold standard.”’ In addition, 
synthesis formats have been placed at the top of what was called the 
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“hierarchy of evidence.”*® They have become the main input to “evidence- 
based medicine,” 
that deal with quality assurance in health care, such as the Cochrane 
Collaboration which essentially produces and disseminates systematic re- 
views about medical interventions.” 

Likewise, the roles of meta-analysis and systematic review in biomedicine 
became a benchmark for other agoras where research and intervention meet. 
With generous references to biomedicine, researchers and practitioners from 
social policy, education or climate science discuss the benefits and potentials 
of these formats for their areas of expertise.'" As a result, systematic research 
syntheses also shaped concepts like “evidence-based policy” and informed 
the work of the Campbell Collaboration or the Collaboration for 
Environmental Evidence, organizations that are very similar to Cochrane. 
Despite the efforts to establish systematic review and meta-analysis in other 
fields, their diffusion was rather limited which is why biomedicine remained 
the only research community with a wider adoption of these standardized 
research formats.'! For this reason, we consider in more detail what makes 
systematic reviewing an evidence practice in biomedicine, and the ways in 
which this practice was established and stabilized. 

In order to shed light on how systematic reviews are attributed to re- 
present evidence, we focus on the exploration of methods and guidelines 
that play a crucial role during the writing of such reviews. Today, sys- 
tematic reviews are a widely accepted subtype of review articles, the latter 
often being perceived as a very diverse and little standardized genre in 
scientific literature.'* Criticism related to the writing of review articles in 
science argued that those would suffer from little methodological control 
and a lack of transparency (e.g. how primary research is included, how to 
evaluate research, etc.).'? Against this background, the systematic review 
article was positioned as a more controlled format of research synthesis, 
where strict and transparent criteria assure the quality of conduct and re- 
porting. Method experts have developed distinct conceptions of what 
systematic reviews are, and how they can be separated from other forms of 
synthesizing research.'* 

As systematic reviews spread among the sciences, their status hinges on 
the agreed upon standard ways of writing and doing such reviews, that is, 
guidelines.'” The epistemic authority of systematic reviews is based on the 
idea that various forms of biases can be ruled out by providing a strict set of 
procedures to follow. Since meta-analysis alone was perceived to be in- 
capable of ruling out various biases, systematic reviewing as a higher-order 
method became pervasive.'® It is against this background that we intend to 
explore the systematic review guideline documents in more detail. In 
steering the author through various stages in the conduct of a systematic 
review, the method promises to reduce the influence of the author’s in- 
dividual preferences and viewpoints. Rather than defining or certifying the 
outcome, the systematic reviewing method defines the practices and steps 
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taken within the process. Therefore, the epistemic authority is based on a 
procedural conception of objectivity, that is, a set of procedural rules which 
authors need to follow if they want their article to be certified as systematic 
review. 

We are focusing our analysis on more recent guidelines for systematic 
reviewing, as these can be best understood as solutions for what was called 
the “new worries of science.” '’ In addition to the sometimes detrimental 
influence of the pharmaceutical industry on the outcome of trials,'? experts 
also criticized how the pressure to publish influences which knowledge is 
reported how and where.” Reporting guidelines such as the “preferred 
reporting items for systematic reviews and meta-analyses,”*' (PRISMA) 
inform the methodologies for reporting systematic reviews, thus the writing 
of the reports. We have therefore chosen to focus our analysis on these 
guidelines, in order to explore the ways in which the renegotiating of 
evidence practice took place in the biomedical field. 

Effectively being a checklist for review authors, PRISMA represents the 
claims for procedural conception of objectivity that is common to sys- 
tematic review methodologies. Yet, at the same time PRISMA seems to be 
different from the rather extensive methodological frameworks for sys- 
tematic reviewing, for example, defined by the Cochrane Collaboration 
which guides the whole process of registering, writing and publishing re- 
views. Different to these large infrastructures, PRISMA consists of a small 
checklist of rules that, if followed by the author, promise to ensure the 
credibility of the decisions made during the systematic review process. 

In contrast to clinical treatment guidelines, PRISMA influences the 
practices of researchers rather than doctors. As such, its successful dis- 
semination and implementation seems to depend on the voluntary accep- 
tance of a wider community, rather than the enforcement of a clinical 
director or a healthcare system. Treatment guidelines may not be accepted, 
and instead create lines of conflict between the social contexts of the 
guideline developers and the guided individuals.” In addition, because the 
making of treatment guidelines incorporates various actors—such as re- 
searchers, practitioners, industry agents, or healthcare officers—the resulting 
definition of a medical practice is shaped by a multitude of interests and 
values correspondingly.” Likewise, actors and corresponding values define 
the evidence practice of systematic reviewing when issuing guidance for 
systematic reviews. Thus, the making and dissemination of PRISMA en- 
tangles characteristics of scholarly communication, evidence practices, and 
practices from clinical guideline making. To understand the processes 
leading to the production of PRISMA guidelines and the justification of 
their status as evidence practices, we address the following questions. 

First, how and by whom was PRISMA constructed, and which factors 
and arguments were negotiated to make it persuasive and acceptable? 
Second, how was PRISMA disseminated in biomedical communities, and 
how has it become so pervasive for authors of systematic reviews? Third, if 
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the social and practical configurations that led to the construction of 
PRISMA mirror the making of medical practice guidelines, how do pro- 
cedural and interventionist paradigms interfere with the prevailing modes of 
evidence practices or academic cultures?* 

We believe that understanding these processes can shed light on how 
evidencing practices are constructed. Interestingly, the case of guidelines for 
systematic reviews provides an example of how instruments initially de- 
signed to guide practitioners outside academia (guidelines) were taken back 
and adapted to what is central for the research enterprise, the art of pre- 
senting evidence in scientific writing. In the next section, we present our 
approach to tackle the questions mentioned above. Presenting and dis- 
cussing our results, we provide five sections that are based on different 
aspects in constructing PRISMA: the creation of a narrative, making the 
guidelines credible, applicable, and explaining how the guidelines were 
disseminated, and implemented. Finally, we present a conclusion based on 
our research questions. 


Methods and Theoretical Background 


To tackle the research questions above, we conceptualized reporting 
guidelines as a specific genre of literature, that lies between typically aca- 
demic and also more performative modes of narrating.” Based on this 
conception, we performed a document analysis to reveal the guideline’s 
rationales and aims as well as the reported methodology that led to its 
construction. To further deepen our understanding, we developed and 
performed expert interviews based on the findings. 

In a first document search, we explored the environment of the 
PRISMA guidelines by identifying relevant publications listed on the 
guideline’s website as well as the website of the EQUATOR network, 
which collects and disseminates various reporting guidelines.” We fur- 
ther used the Web of Science bibliometric database to identify all 
guideline updates, translations, and guideline forks. Focusing on the main 
PRISMA publications, we performed an exploratory document analysis 
to study not only its bibliographic but also the textual characteristics.” 
This analysis was guided by a framework which focused on issues of 
scholarly communication, such as publication formats, authors, referen- 
cing, or number of pages. 

Key to understanding guidelines such as PRISMA as a genre of their own 
is the idea that they offer a specific and shared set of communicative pur- 
poses which are different from that of traditional research articles.”® First of 
all, guidelines appear to be more technical than other forms of scientific 
writing, providing a list of rules to follow. Yet for these rules to be followed 
and have an impact, guideline publications needed to be granted legitimacy 
by community members.”’ Therefore, guidelines also contain textual ele- 
ments designed for persuasion, that is, a narrative or rationale why 


106 Alexander Schniedermann et al. 


guidelines and standards are needed and why the concrete effort appears to 
be plausible. For this reason, the document analysis focused on exploring 
the ways in which the guidelines and the process of creating them were 
designed to create academic credibility. 

To further deepen our understanding of what has been found in the 
document analysis, we planned and performed semi-structured interviews 
with six guideline experts during the first quarter of 2021. Interview par- 
ticipants were sampled from authors, translators, or workgroup members of 
one or more of the PRISMA publications or its updates. While interviews 
have been anonymized for ethical reasons, the interviewees have been 
asked to provide substantial information about their role in the develop- 
ment process. Analyzing the interviews, we used deductive coding based on 
the key themes identified in the document analysis as well as theoretical 
accounts of standardization, the construction of clinical practice guidelines, 
and communication in academic communities. 

Each of the interview participants had different roles in the making of the 
earliest version of the guidelines—QUOROM—or one of its later updates. 
Two were involved as authors in the 1999 version (Participants B and C), 
and its 2020 version (Participants E and F), while only one authored the 
2009 version of the guidelines’ explanatory document (Participant A). Most 
interviewees were part of the expert group that developed PRISMA, yet all 
of them were researchers active in various fields, such as information re- 
trieval, research ethics, clinical research, or research design. In the 2009 
version of PRISMA, two participated in the guidelines (Participants A and 
B) and one also contributed to the explanatory document (Participant B). 
Although many workgroup members were listed as authors in the 2020 
update, two interviewees remained mere workgroup members (Participants 
A and B). While all of the interviewees have at least some background in 
the biomedical sciences, one participant is especially known for his work as 
an editor of an academic journal (Participant B). Several are also associated 
with the Cochrane Collaboration (Participants A, B, and F). In addition, 
one interviewee has substantial experience in the medicine related industry 
(Participant C). 

Our analysis is guided by a framework that focuses on “the multiple ‘worlds’ 
of a guideline.”*” It consists of four “worlds,” or repertoires, that express the 
discursive forces in the construction of guidelines. First, the repertoire of sci- 
ence consists of argumentative strategies about the identification and appraisal 
of relevant evidence and mainly comes in the form of systematic reviews or 
meta-analyses.°! Second, the repertoire of practice, which attempts to link a 
guideline text to the practices in a clinic in order to evaluate its usefulness. 
Third, in the repertoire of politics, guideline developers envision different 
stakeholders to discuss the acceptability of the included statements, and how 
the guideline embodies political positions that may affect power relations. Last, 
the repertoire of process understands the group as an apparatus of knowledge 
generation, which constructs guidelines by employing a reliable methodology.” 
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By identifying such dimensions in the creation of PRISMA, we can analyze 
substantial similarities between the construction of clinical practice guidelines 
and reporting guidelines for researchers. 

However, we must consider the substantial differences between clinical 
practices and evidence practices in scholarly communication, and inform 
the analytical framework proposed above accordingly. Different to other 
fields, biomedicine has a well-established set of research techniques which 
lead to determinable and replicable outcomes. When facing new research 
problems, biomedical researchers can rely on an agreed upon set of practices 
and tasks to establish knowledge claims. This low level of task uncertainty is 
characteristic of what Richard Whitley has termed “professional ad- 
an organizational configuration of professions which are 
characterized by their degree of division of labor. We are interested in how 
these task uncertainties are reduced and accepted due to technical stan- 
dardization and formal training. Such standards enable the communication 
between distant communities by changing local disciplinary practices in 
relation to overarching goals and aims.** But how does such standard- 
setting take place? 

Scientific communities are structured by disciplinary authorities that define 
standards and practices in their local domain.” In order to retain their status, 
such authorities eventually become more resistant to overarching standardi- 
zations. In comparison to other scientific fields, biomedical communities in 
particular are more fragmented into local centers of authority that have to 
negotiate new forms of standards on a dynamic level.” In recent years, 
biomedicine and its subfields have also been influenced by the strengthening 
of role models to act more autonomously in negotiating priorities between 
research and application—or bench and bedside.” For this reason, estab- 
lishing standards has become more complicated and requires more rhetoric 
and discursive effort, in order to not only to be accepted by researchers, but 
also to compete against other potential forms of standardization, for example, 
provided by the Cochrane Collaboration. In addition, standard conflicts may 
not be avoided or resolved by overarching or central authorities, as in the case 
of clinical practice in which checklists and treatment guidelines are im- 
plemented by clinical directors or healthcare policies.** Rather, the standard 
must be made highly compatible with existing and agreed upon disciplinary 
as well as local regulatory authorities, for example editors of academic journals 
as the gatekeepers of scientific fields.*” 

Given these theoretical considerations, we extended the “multiple 
worlds framework,” proposed by Tiago Moreira, to fit the specific char- 
acteristics that can be found in academic practices and scholarly commu- 
nication. First, we put a much stronger emphasis on how the standard 
employs a narrative for the profession to provide rationales for application 
and build the “repertoire of science.” Second, we focused on the role of 
academic journals in disseminating the guideline, for example publishing 
the guidelines as well as enforcing it by implementation. And we further 
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investigated how a “repertoire of journals” also influenced the creation of 
the guidelines in order to fit it into the role of the gatekeepers of science. 


Results and Discussion: Invoking the Crisis 


PRISMA attempts to influence the practice of writing systematic reviews. 
It is dependent on making authors aware of the genre’s shortcomings, and 
creating acceptance for change. To understand how PRISMA constructs 
such acceptance and calls for change, we analyzed its argumentative stra- 
tegies and their evolution. Essentially, in order to persuade readers, the 
guidelines establish a narrative which makes the practice of systematic re- 
viewing problematic. 

PRISMA creates a story about the current state of systematic reviewing, 
the problems, and the potential solutions—of which the PRISMA guide- 
lines are only one. As such, they not only construct a profession and 
“enroll” the reader into their reasoning,*” but also employ an operational 
mode that calls for action.*' The professional story unfolds with the role of 
systematic reviews and meta-analyses for contemporary biomedicine: 


Systematic reviews and meta-analyses have become increasingly im- 
portant in health care. Clinicians read them to keep up to date with 
their specialty, and they are often used as a starting point for developing 
clinical practice guidelines.*” 


Beyond iterating the functions and epistemic promises of systematic re- 
views, this story also creates narrative links to the actors who value and 
promote them to shape a community." While guideline developers, 
doctors or healthcare systems are mentioned in all versions of PRISMA, in 
its first iteration (the QUOROM guideline), it also mentions the Cochrane 
Collaboration.“ 

In a second step, the guideline texts explain the various flaws in the 
reporting of systematic reviews. In doing so, the text refers to the common 
variance in the quality of scientific publications, rather than accusing sys- 
tematic reviewing or review authors more explicitly. 

“As with other publications, the reporting quality of systematic reviews 
varies, limiting readers’ ability to assess the strengths and weaknesses of 
those reviews.” In addition, the text refers to several observations to 
support these claims, and persuades the reader that applying the guidelines 
will provide a viable solution to the presented problems. In citing studies 
that either prove the lacking reporting quality of reviews, or evaluate how a 
guideline can improve reporting, the PRISMA document combines two 
strands of research into a new story.“ Utilizing the repertoire of science, it 
gathers the support of several researchers and their studies to become a 
technical document itself. This makes rejecting its statements and 


On Top of the Hierarchy 109 


conclusions more tedious, since critics have to reject all supporting refer- 
ences and claims.” 

Each version—QUOROM in 1999, and PRISMA in 2009 and 
2020—witnessed roughly a decade of development in systematic reviewing. 
While the main rationale and several arguments can be found in each version 
in a similar way, their differences relate to the genre’s role for scientific 
communication. With QUOROM as its first version,*® the narrative of the 
responsible profession was built particularly around meta-analyses—the sta- 
tistical aggregation technique—even though the guideline mentions the 
conceptual differences between meta-analysis and systematic reviewing by 
referring to the Potsdam consultation held in 1994, which was one of the 
earlier international gatherings to discuss the state of research syntheses:*” 


Several queries addressed the distinction between the meta-analysis and 
systematic review. As we indicate in the introduction, and throughout 
the statement, the QUOROM group agreed to observe the distinction 
as defined by the Potsdam consultation on meta-analysis.” 


In the later versions this was changed.”' A more inclusive wording was used 
to explicate a wider methodological scope and applicability. Besides re- 
naming the guideline from QUOROM to PRISMA to actually include the 
words “systematic review,” its narrative sections now mentioned how the 
systematic review has become important to actors other than healthcare 
decision makers or researchers. It adds guideline makers, clinicians, funders, 
and even journal editors, and thereby claims that the guidelines’ role in 
changing practices is relevant to a wider array of publics and communities. 

The updates in 2020 renew the original intentions and target audiences 
but still mention wide applicability, even for nonmedical practices such as 
“social or educational interventions.”°” Although its narrative employs an 
overall more neutral tone, it distinctively establishes links to other actors 
that gather around the guidelines. Most notably, it mentions the wide array 
of PRISMA extensions that modify the 2009 version in order to account 
for specific review methods, study types, or disciplinary specialties. In ad- 
dition, the document refers to the EQUATOR network of reporting 
guidelines, and explains its compliance with the network’s guidance on 
creating reporting guidelines.” It thereby stabilizes the PRISMA guidelines 
by situating them in a network of standardization organizations. 

Similar to how the narratives within clinical guidelines evolved from 
healthcare decision making to other uses of knowledge syntheses, guideline 
developers aligned PRISMA’s story with current topics in the research 
community. Most notably, several participants elaborated on the relation 
between PRISMA and the problem of reproducibility or replicability of 
systematic reviews.°* Subordinating the problems of systematic reviews 
under those “new worries of science,” and with PRISMA as a weapon in 
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the “credibility revolution,”°° the overall endeavor is equipped with sub- 
stantial intellectual value and societal legitimization. 

The stories, and their evolution during the development of PRISMA 
and its updates, construct multiple professions. In the early versions, a di- 
verse set of actors such as researchers, practitioners, or policymakers are 
narrated into a community of users and producers of systematic reviews. 
Furthermore, PRISMA’s terminology and selection of items for inclusion 
shapes this community’s genuine understanding of what a systematic review 
is. Framing the epistemic crises in and around systematic reviewing, the 
story fuels the quality assurance movement in contemporary biomedicine. 
This consists of academic researchers and publication experts, who address 
the problems of reviews by scientifically evaluating the genre. Similar to 
how traditional review articles narrate topics, assumptions, and results into a 
scientific field,” the guidelines connect topics, arguments, studies, and even 
institutions. How this profession was shaped into an active discourse 
community will be elaborated in the next section, in more detail. 


Guidelines in the Making 


In order to be granted legitimacy, the guidelines had to present the process 
by which the different items relevant for the practice of systematic re- 
viewing, the contributing guideline authors, and the description of the tasks 
were articulated. As such, the content of the PRISMA guidelines and its 
updates were drafted and discussed in several meetings, surveys, and con- 
ferences. Based on interviews with the guideline authors and document 
analysis, we will now further elaborate on the processes that shaped the 
making of PRISMA. Beside the role of the steering committee, the for- 
mation of expert groups, and formal consensus practices, the enrollment of 
various actors effectively fostered the dissemination of the guidelines and 
initiated a network of what might be considered as PRISMA’s own pro- 
fessional community. 

We first analyzed how, and in what ways, experts were invited, and what 
role they played in convincing communities to change their practices. The 
core of the guidelines was formulated by a small group of actors. A central 
role was taken by the steering committee—as the method sections of the 
guidelines and our interviews have revealed. As the first iteration of 
PRISMA was built from scratch, this committee collected available 
knowledge about reporting, and drafted the first items. It was this group 
which also coordinated later revisions or versions of the guidelines. 
Although the updates were built upon each other and involved additional 
actors, a few core experts still led the development by writing substantial 
parts of the texts, and dealing with comments. Recalling experiences with 
other guidelines, interviewees stressed how the steering committee’s efforts 
can benefit or harm the overall workflow: 
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When the team is very capable, they would have done a lot of prep 
(aration) work ahead of time, before they engage with you to solicit 
input on the specific things. When the team is less capable ... it’s going 
around and around and you never get to what needs to be done or how 
this can be done in a way that not only incorporate most people’s 
feedback but also efficiently ... . in terms of time-wise and how many 
rounds of revision we need.” 


Crucial to the construction of PRISMA were the involved experts. Starting 
with 29 contributors in 1999, growing to 42 in 2009, and finally to 139 in 
2020, PRISMA and its network grew substantially over time. Similar to 
what Latour has called “bringing friends in” to explain the argumentative 
force of referencing,®’ the number and status of the involved experts 
provides the guidelines with intellectual and social authority. Since most 
intended recipients and readers of PRISMA are authors of systematic re- 
views, the group must achieve a proper representation of disciplinary and 
methodological plurality to avoid conflicts with local authorities, such as 
prolific specialists or groups that usually establish disciplinary practices and 
standards: 


So we did the first guideline and it was an interdisciplinary group of 
people. There were statisticians. There were trialists, people who run 
randomized controlled trials. And perhaps most importantly, we had 
some influential journal editors.°! 


Authoring a guideline document such as PRISMA can boost one’s aca- 
demic impact, due to the high citation rates of this genre. In addition to 
selecting experts based on their skills and roles of contribution, selection 
criteria must warrant the expert’s proper motivation to participate: “As you 
probably know, these guidelines are highly cited. People want to have them 
on their CVs. It’s beneficial to your career. Sure, if you have something 
that’s been cited a thousand times.”°? 

Analyses of each individual author’s affiliation reveal that these were 
carefully selected, each representing different fields of expertise. Negotiating 
and defending their boundaries against other experts, prolific researchers take 
a central role and make local authority claims, for example, about valid 
methods or canonical interpretations. By agreeing or disagreeing on the 
narratives, other disciplinary researchers gather around these new cores, and 
build social structures and networks.°* The disciplinary experts who con- 
tributed to the new standard can transform the perception of who (or what) 
the local authorities are. The local integration of PRISMA devalues sys- 
tematic reviews that do not comply with this standard. Therefore, as with 
other standardizations, the negotiation and integration of PRISMA may 
reshuffle authority claims in biomedical disciplines. Yet, according to our 
interviewees, the effects of guidelines are often less glamorous than expected 
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though we have indication from our fieldwork that PRISMA has achieved to 
become a particularly reputed way of reporting systematic reviews, especially 
when it is planned as a collaborative endeavor. 

But the selection of experts focused not only on the intellectual con- 
tribution of experienced researchers but also on the effective dissemination of 
the guidelines. By bringing experts from different fields into the process of 
formulating the guidelines, the guidelines become related to these different 
fields of which these authors were part of. Therefore, the representation of 
targeted groups is crucial for the acceptance and dissemination of guidelines, 
as it creates awareness and supports “marketing.” The involvement of 
journal editors particularly served this role, as one interviewee pointed out: 


I remember we would be looking at who were the key stakeholders 
and journal editors were key with the view that both you’ve got the 
voice of the journal editor influencing [the] guideline, but you’ve (also) 
got the ability for other journal editors to say, “Oh, somebody of my 
type was involved. Maybe I should pay more attention to it.” And then 
it was a bit of lobbying of the journals to say, “This is a good thing to 
be doing. It will help your transparency.”°’ 


By establishing cross-disciplinary narratives and enrolling various dis- 
ciplinary authorities, PRISMA can be disseminated and implemented into 
several biomedical subcommunities. Furthermore, since there is no central 
authority that issues regulations and standards—setups that can be found, for 
instance, in clinical practice—guidelines must enroll and cooperate with 
local authorities in order to become pervasive.®* In representing the shared 
efforts of such connected local authorities, the PRISMA documents es- 
tablish the network of group members and their institutional affiliations. To 
display this expert group, members were turned into a composite author 
called “the PRISMA group,” that was listed as last author.” In contrast to 
merely listing the contributors’ names in the acknowledgement, this rather 
uncommon type of authorship stresses that members contributed in mul- 
tiple ways in addition to the writing of text. Yet at the same time, the text 
employs a personal tone which makes not only the authors visible but also 
the wider group.’” 

How the different experts collaborated with each other on PRISMA is 
outlined as a method section in the guideline document. This establishes a 
textual link between the procedure and the resulting guideline items. 
Additionally, the resemblance to other reports of empirical research presents 
guideline making as a research process itself. As such, PRISMA attempts to 
create causal links between the group meetings and the resulting rules for 
reporting." By unfolding and communicating this link, the text of the 
guidelines provides the repertoire of process, and attempts to understand 
the making of PRISMA as a reliable methodology to negotiate and consent 
on a proper set of rules.” Therefore, the social configuration of the expert 
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group as well as its communicative processes become an abstract recipe that 
contains mechanisms that build trust and acceptance by the biomedical 
community. 

Turning the construction of PRISMA into an abstract procedure, its 
techniques were related to other agreed upon methods or standards. This 
increases the transparency and acceptability of the PRISMA standard, since 
authors can understand the standard’s connections to their local practices.”° 
For example, the guidelines and the interview respondents both mentioned 
the prominent role of the Delphi methodology to achieve formal and re- 
liable consensus. Promoted by the United States’ National Institutes of 
Health (NIH) during the 1970s and 1980s, it has become an established 
procedure to organize expert consensus and mitigate individual biases.’* In 
another example, the PRISMA authors had to comply with the authorship 
standards of the International Committee of Medical Journal Editors, by 
omitting “the PRISMA group” as an author in the 2020 version. 

Unsurprisingly, guideline construction procedures became standardized 
themselves. Influential guideline experts have formalized this procedure 
into “guidance for developers of health research reporting guidelines,” ”” in 
essence a guideline for creating guidelines, or a practice that constructs 
evidence practices. As interview participants noted, the PRISMA group 
was eager to standardize their own communication procedures and con- 
tribute to such a guideline. Recalling efforts by one of the leading experts, 
one interviewee mentioned: 


I mean there is standardization at the root of the guideline, but then it’s 
the process and getting to that guideline, that also is its own 
standardized process that I think he’s trying to standardize because 
there are many different approaches. ’° 


In turn, the latest update of the PRISMA guideline was based on this now 
standardized procedure and also reports its compliance with it. Such “network- 
building” provides a mutual legitimization of both standards—the guidelines 
for reporting reviews, and the guidelines for creating guidelines—and makes 
them more authoritative.” Turning its own construction even further into a 
procedurally regulated endeavor, PRISMA benefits from the same valuations 
as systematic reviews that comply with PRISMA. Since PRISMA turns 
practices into evidence-practices by the procedural inscription of values, 
guideline making has also been transformed into an evidence-practice. In other 
words, after redefining how knowledge turns into authoritative evidence, the 
guideline can become evidence by the very same definition. 

Beyond the construction and interaction with guideline objects, developers 
further professionalized the role and authority of guidelines by systematically 
evaluating their effectiveness. Titles such as “Epidemiology of systematic re- 
views,”’® witness how epidemiological methods and concepts were turned 
into a new professional narrative, coming from “meta-epidemiology,””” into 
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more interdisciplinary conceptions of “meta-research.”*° Leaving the bound- 
aries of epidemiology aside, interdisciplinary researchers, institutes, journals, 
and educational programs focused on providing the “the repertoire of ‘sci- 
ence” in guideline development.” Aligning these tasks with the overall goal of 
moving toward higher quality science and evidence-based medicine, the 
network became a constituency that not only benefits from various expertise, 
but also from the representation of different groups.” 

As we have shown in this section, the steering group attempted to manage 
what has been called the “second tension” throughout this book. Although the 
definition and evaluation of guidelines is heavily influenced by scientific 
practices, the formed constituency provides a cross-professional space where 
review authors, journal editors, and guideline researchers are invited to con- 
tribute. This helped to make the issues of PRISMA visible and valuable to 
those various contexts. At the same time, the possibility to equally influence 
the logic of the guidelines avoids the impression that one group or field au- 
thoritatively translates its knowledge to another. Thus, potential frictions be- 
tween the diverse groups were actively minimized throughout the process. 


Applicability by Simplicity 


In order to influence writing practices, a guideline needs to formulate 
concise rules which authors can follow. In this section, we deal with how 
these items are formulated and—based on the interview results—what 
motivated the guideline authors to do so. 

PRISMA complies with the typical form of a journal publication. As such, 
it not only provides a structure consisting of a narrative and methodology 
section but also presents its regulatory items like research results. First, in a 
sample flow chart, it displays how authors of systematic reviews can properly 
report the stages of including and excluding primary studies. Second, the 
item list displays the various aspects and rules of proper reporting, which may 
be used as a checklist by authors. It grew from six overarching categories that 
roughly address the stages of reviewing,*? to 27 separate and rather detailed 
items in PRISMA’s 2009 version.®* With the latest update, some of the items 
were given additional subitems so that the overall number did not change in 
the 2020 version. As interviewees have noted, much of the efforts during 
item formulation negotiations were related to stripping down the number of 
items and rules so that the guidelines would not only be publishable, but also 
easy to use. Not surprisingly, some respondents have explained that PRISMA 
represents only the minimum or mandatory reporting, rather than what 
could be considered as recommended or optimum. 


Because things were discussed (that) we couldn’t possibly put everything 
in. So we had sort of an outline about which would be the required 
items in the checklists. And that’s what the majority of the time the in- 
person meeting was involved with, then we all went away.” 
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In its focus on PRISMA’s applicability, the group took a different approach 
to other attempts in standardizing the methodological quality of reporting 
or conduct. Especially in the case of systematic reviewing, potential 
guidelines compete against the epistemic and social authority of the 
Cochrane Collaboration. By providing extensive and strict methodological 
guidance for conduct and reporting, software infrastructures, detailed edi- 
torial supervision, and publication in its own database, the Cochrane 
Collaboration defines the overall process of systematic reviewing on a much 
more comprehensive level.*° For this reason, interviewed participants not 
only mentioned the greater quality of Cochrane reviews in conducting and 
reporting but also the much higher efforts to perform and write them. 
Thus, the group also implicitly focused on applicability by juxtaposing their 
goals and efforts to those of Cochrane. 


Cochrane is very detailed, and there’s Cochrane for every kind of thing 
that you want to do. It was clear from the get go, that we were not 
going to be a Cochrane group, we were not going to hang together for 
the rest of our careers doing this.” 


Limiting the size and scope of guidelines is a common phenomenon in the 
construction of clinical treatment guidelines. Since the guideline has to fit 
into its applicatory context, the group tries to envision the “repertoire of 
‘practice,’ which refers to the usefulness of the guideline.”** Developers 
weigh the guidelines’ knowledge claims against the circumstances of dif- 
ferent contexts and usage cases in order to judge whether the guidelines will 
be useful and improve the overall outcome. In contrast, extraordinary si- 
tuations may demand improvisation and are not covered by the guidelines’ 
knowledge claims, rendering it inapplicable or useless. Envisioning such 
cases enables developers to configure the standard’s complexity and scope of 
application. 

A careful consideration of complexity and scope is particularly important 
for the dissemination of multidisciplinary guidelines, such as PRISMA.*? 
Highly applicable guidelines provide substantial support in most situations 
and practitioners perceive them to be useful, so that they willingly accept 
the guideline. In addition, translations into different languages increased its 
applicability among nonacademics, especially medical practitioners, as one 
interviewee noted.” Therefore, what some interviewees perceived to be 
“the minimum,” may be better interpreted as the most cost-effective level 
of reporting, in which the effectiveness of each rule is weighed against the 
necessary efforts to comply with it.”! 

Seen in this light, PRISMA’s particularly restricted focus on reporting 
makes it applicable as a standard for systematic reviews of various study 
types and research topics. In contrast to Cochrane’s standard, which pro- 
vides guidance for the design and conduct of reviews, PRISMA explicitly 
regulates only the writing of reviews, and tells authors what details have to 
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be included in their manuscript. Although the distinction between conduct 
and reporting is fuzzier in the case of reviews than in clinical trials, the focus 
on reporting avoids conflicts with methodological plurality, as participants 
explained. In that sense, the guideline limits its own scope and controls the 
number and size of targeted groups, which makes potential conflicts 
manageable.’” However, the restriction to reporting also mobilizes the 
regulatory capabilities of academic journals, as we will explain in the next 
sections. 


Dissemination by Journals 


The guidelines were published in biomedical journals in order to effectively 
reach potential authors of systematic reviews. In this section, we discuss 
some of the unique characteristics of its publications, found during the 
document analysis and the interviews. 

The PRISMA documents were published in multiple scientific periodicals. 
While QUOROM was published only once,” the 2009 version was pub- 
lished in seven different journals, ”* and was officially translated three times,” 
in order to quickly approach multiple biomedical sub-communities. Likewise, 
the explanation and elaboration document—an additional paper that offers a 
more detailed look into the guideline’s items and also provides some 
examples—was published in five different journals in 2009 and translated 
once.” The latest update has been published as a pre-print, and in five dif- 
ferent journals.” In addition, there was one additional explanatory docu- 
ment.» Since cross-publication is usually considered unethical for researchers, 
interviewees noted how it was suggested and discussed by the team’s pub- 
lication experts. 

Beside the potential for faster communication, cross-publication also 
promised wider access, since researchers and their respective institutions 
may have different journal subscriptions. But it also means the involvement 
of multiple editorial offices and peer review processes, which demands huge 
efforts from the guideline developers. In addition, different editorial stan- 
dards or peer reviews can lead to huge variations in the final documents, 
although all are intended to represent the same standard.”” More strikingly, 
since the guidelines already incorporate the intellectual contributions of 
many experts, peer reviews may undermine the sophisticated consensus 
practices. As one participant noted: 


What I would call the aftermath of that ... was a general sense of, 
“we're not going to do this again.” Why are we publishing in so many 
journals the same thing? And it is one of the challenges with publishing. 
That means what are the journal(s) supposed to do about peer review, 
given that so many dozens of people have been involved in reaching 
this consensus? How can any of that be changed by one or two peer 


- > 100 
reviewers: 


On Top of the Hierarchy 117 


The guidelines were placed among some of the “big five,” 1°! which are the 


most impactful journals in biomedicine. While QUOROM was the only 
document in The Lancet, PRISMA of 2009 and its 2020 update were 
published in the British Medical Journal and Annals of Internal Medicine. Other 
journals were PloS Medicine, the Journal of Clinical Epidemiology, the 
International Journal of Surgery, Systematic Reviews, Open Medicine (dis- 
continued), Physical Therapy, and the Italian Journal of Public Health. In ad- 
dition, four official translations are listed on the PRISMA website, of which 
three were also published in national journals. 

In the interviews, participants argued that the group targeted high impact 
journals, as these have more resources for methodological quality assurance, 
and the necessary awareness for improving reporting. But more im- 
portantly, such journals improve the dissemination, since they are more 
central and authoritative within their respective communities. High impact 
can be achieved by specialized journals that have become local authorities 
within usually smaller sub-disciplines. On the other hand, generalized 
journals often reach high citation impacts too. Since such generalized 
journals target various audiences or even link many sub-disciplines, their 
published methods and standards are subject to intense criticism and aca- 
demic competition.'°” Therefore, the successful enrollment of more gen- 
eralized journals into PRISMA’s narrative provides the guidelines with 
cross-field legitimization and authority. 

In publishing and implementing PRISMA, a journal can make an in- 
dividual commitment toward reporting quality in its domain. The journal 
employs the professional story of the guidelines, and aligns itself with the 
experts and networks that developed the guideline; whether they are from 
the same domain as the journal or not. But similar to the guideline de- 
velopers that contributed mainly in order to boost their academic re- 
cognition, journals benefit from PRISMA’s high impact too. Not 
surprisingly, interview participants noted that cross-publication was 
brought up by journal editors, and some of them might have been moti- 
vated by improving the impact metrics of their periodicals. Similar to 
scholars who can shape their epistemic practices in accordance with such 
metrics, °’ journal editors try to influence their metrics by inviting authors 
or to solicit review articles.'°* Likewise, journal editors can participate in 
the development and dissemination of highly cited guidelines and standards. 


Enforcement by Gatekeeping 


In contrast to other standard innovations, PRISMA would also be im- 
plemented into editorial processes, so that authors of systematic reviews 
would have to comply with PRISMA in order to get their manuscript 
accepted. In this section, we discuss the extent and role of this practice. 
Usually, new standards or methods for “doing science” become established 
by orchestrating and demonstrating their epistemic superiority, so as to 
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finally persuade individual researchers. Whereas evaluation and demon- 
stration of effectiveness ordinarily requires time and additional resources, as 
explained above, PRISMA was legitimized with the help of the partici- 
pation of prolific researchers and, as will be shown, academic journals. 

Traditionally, individual researchers demand more evidence for a stan- 
dard’s effectiveness before they comply with, and this can hinder quick 
acceptance and dissemination. Many methods or standards become 
pervasive only due to traditional modes of academic quality assurance; 
largely, peer review. Since, in a peer review system, experienced and 
prolific scientists evaluate the intellectual contributions of other researchers, 
they can utilize whatever standard or guideline they find appropriate. 
Although this system generally ensures a certain level of academic quality 
control, it notably did not prevent the decrease in reporting quality in the 
first place. 

However, peer reviewers may now consider PRISMA when they ap- 
praise the reporting quality of systematic reviews, or when editors ask them 
to do so. But this still means that the dissemination and application of 
guidelines is dependent on the awareness, and application by many in- 
dividuals. So, varying expertise in relation to reporting does not only result 
in authors submitting incompletely reported manuscripts, but also peer 
reviewers not detecting the flaws. 

“Unfortunately, you’re assuming everybody is at (a) certain level. You 
are assuming people would know how to write a paper, but that’s not the 
case.” 16 

In addition to the role of peer reviewers in the evaluation of manuscripts, 
editorial offices also have substantial influence on deciding whether a 
submission will be published or not. Usually, editors decide which sub- 
mission will be sent for review, and select appropriate peers, which is why 
they are called the “the gatekeepers of science.” '°’ They determine the 
intellectual contexts of a submitted manuscript and also heavily influence 
formal characteristics of the texts, such as writing styles or references. °’ In 
fact, interviewees have mentioned that editorial offices may even have 
aggravated the reporting crisis by putting limitations on word or reference 
counts. Nevertheless, guideline developers hoped that PRISMA would be 
implemented into journal processes and overseen by editorial offices or 
method editors. 

Ideally, authors would fill out the checklist and submit it as additional 
material together with their manuscript. The editorial office or specifically 
trained methods experts would then appraise the manuscript and the 
checklist in order to judge the level of compliance. Due to the procedural 
nature of PRISMA, compliance can be checked on a per-rule basis, which 
turns the overall handling into a more formalized process that can be 
performed without proficiency in either the content or method of the 
review. Such a process could even be automated, similar to the checking of 
statistical reporting. "° Utilizing editorial capacities in advance of the peer 
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review system would provide the necessary centrality to ensure that every 
systematic review is judged by the same criteria. This increases the authority 
of PRISMA by extending its intellectual authority with a more formalized 
type of regulation enacted by the gatekeepers of science. 

Beyond the publication in high impact journals, several journals officially 
endorsed the guideline. As the 2020 version concludes, PRISMA was 
endorsed by almost 200 journals or organizations publishing systematic 
reviews, which provide the guidelines with the necessary outreach to 
achieve a wide dissemination. However, the mere endorsement does not 
clarify the actual level of implementation which can vary a lot. So, while 
engaged editors verify the submitted checklists and occasionally ask authors 
for further clarifications, other endorsers may just publish them together 
with the manuscript.''° 

As our interviews have shown, the goal of making PRISMA im- 
plementable by academic journals played a crucial role in the making of the 
guidelines. As Tiago Moreira suggests, guideline makers discuss the ac- 
ceptability of the guidelines by assessing the “repertoire of ‘politics.””''' In 
doing so, they envision the guidelines’ recommendations through the lenses 
of the various groups and identify potential lines of conflict. In interpreting 
the relations between authors and journals, the PRISMA group imagined 
the latters’ regulatory capacity and decided what editors can demand from 
authors, before they withdraw their submissions and turn to a competitor. 
As already indicated above, the often emphasized boundary between 
conduct, as performing the review, and reporting, as writing down the 
results, reflects the careful appraisal and utilization of these regulatory ca- 
pacities. In other words, PRISMA was tailor-made to be implemented on 
the journal level, as participants explained: 


Journals are able to implement a guideline by way of telling authors 
who must adhere to this guideline, it’s a lot easier to get to authors and 
researchers that way. I think that’s one of the key reasons from my 
understanding as to why reporting guidelines have been the focus over 
conduct guidelines. 


Reducing the guidelines to a very specific set of values and requirements 
enables the journal to establish it as a required standard. Although academic 
journals have achieved some level of authority independent of individual 
experts or the overall referee system, they are subject to various epistemic 
and social constraints.''* As such they must navigate between the economic 
expectations of the publishers, and authors who might submit to a different 
periodical if the imposed formal requirements become too burdensome or 
not applicable to their research.''* Similar to when clinical guidelines are 
neglected whenever anomalies occur during medical treatments, authors 
can turn to a different journal when their systematic reviews are of such a 
type that PRISMA is not applicable. In this respect, the spread of PRISMA 
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is more like standardization in the industry, in which standards are issued by 
a variety of competing organizations that develop and market their stan- 
dards as a form of decentralized regulation or soft law. t? 


Templating Evidence in Biomedicine 


In this study, we tried to understand how biomedicine’s most appraised 
evidence practice, the writing of systematic reviews, is shaped and defined 
by formal standards, most notably the “preferred reporting items for meta- 
analyses and systematic reviews,”''® or PRISMA. Understanding this 
reporting guideline as an attempt to standardize evidence practices, we 
investigated how texts, researchers, methodologists, and journal editors got 
engaged in social configurations and practices to create a guideline that is 
applicable, and acceptable. To do so, we performed a document analysis of 
the PRISMA reporting guideline, and interviews with its developers. By 
using the “multiple worlds” framework suggested by Tiago Moreira,''’ we 
identified several similarities and differences between the construction of 
PRISMA and clinical treatment guidelines. 

The emergence of PRISMA exemplifies the interaction of medical 
practice and biomedical research, described as the “second tension” 
throughout this book. In telling a coherent story about the problems of 
systematic reviewing and how those can be solved, PRISMA focuses on the 
practices of reviewing and how those can be improved procedurally. 
Conceived as a regulatory tool, PRISMA represents an intervention aimed 
at improving reports of systematic reviews so that they can be considered as 
biomedical evidence. This originates from what is often understood as the 
core of evidence-based medicine, in which doctors are provided checklists 
and standards to increase treatment uniformity and reduce errors.'!® 
Likewise, PRISMA tries to guide authors through the writing of systematic 
reviews to ensure that all reviews contain the necessary information and 
nothing is left out. So, the basic principles behind the improvements of 
medical practices were used to improve evidence practices such as sys- 
tematic reviewing. 

In becoming a new standard, PRISMA interferes with already established 
practices and local authorities that have defined the characteristics of sys- 
tematic reviews. In our analyses, we found that some efforts were taken to 
make PRISMA applicable, as well as acceptable. Besides keeping the 
guidelines rather simple for easy application, it was designed with respect to 
the regulatory capabilities of academic journals. Since PRISMA focuses 
only on reporting, compliance can be enforced or supervised by editorial 
offices. This turned the guideline not only into a regulatory tool for journal 
editors, but also fostered a wider dissemination and application. In addition, 
its developers formed a professional community that evaluates endorse- 
ments and compliance with PRISMA. This community also provides 
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updates and extensions in order to keep PRISMA relevant to contemporary 
trends in biomedical research. 

Our analysis has shed some light on the configurations and decisions that 
enabled the wide dissemination and application of the PRISMA guideline in a 
diverse and global endeavor that lacks in central authorities. As such, this study 
provides insights into standardizations and the very mechanisms that shape evi- 
dence practices in science. Since disciplines such as the social sciences, psychology 
or climate science also proclaimed crises, our results can be used to inform similar 
efforts in those fields, or at least, explain if and when such forms of overarching 
standardizations are inapplicable.''? However, more research is needed to better 
understand some factors affecting the dissemination and application of PRISMA, 
as well as the reflexive discourse that follows its implementation. 
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